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ABSTRACT 

Examples of cross-cultural stability or instability 
of mental test items are illustrated. A statistical procedure 
involving the cross- plotting of item difficulties for two different 
groups and generating a line of mutual regression through the 
resulting scatter of points is described. D-values, representing the 
perpendicular distance^ in delta units, from the liii:^ of mutual 
regression, for all items in a given subtest {in thxs: case, 
mathematics) ore presented in a computer generated taMe for the 
following groups studied: American Indian, Af ro^^Amer.i oan, 
Mexican-American, Puerto Eican, other Latin-American, 
Oriental-American, White Eastern, White Southern, and White Western, 
( Author/::^C) 
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AN INVESTIGATION OF CROSS-CULTUKAL STABILirf 
IN MENTAL TEST ITEMS ^ 

Hunter M. Breland 
Educational Testing Service 

For purposes of illustrating what I intend by cross-cultural 
stability (or instability) in a mental test item, I want to present 

some examples. Imagine that you are back in the first grade and 
the teacher is administering a test to determine what you understand 
about v7ords. A booklet is given to you and each page has two columns 
of pictures. The teacher instructs you to mark with a big X either 
the picture in the left hand column or the picture in the right hand 
column as she pronounces a word representing one of the pictures. One 
of the picture-pairs has on the left a common device for use with 
baby diapers and on the right a common writing implement, as shown at the 
top in Figure 1. Now imagine that I am the teacher. I will pronounce 
the word and you are to put a big X across the picture that the word repre- 
sents. Are you ready? (I pronounce the word). Most of you would mark 
the picture on the left because, in your socio-cultural group, the word 
representing the picture on the right is pronounced differently than the 
word representing the picture on the left. In the southern part of the 
United States, however, the words are pronounced the same. The two pictures 
are distinguished by preceding the "common" word by another word. That is. 



1. This paper is based upon results obtained in a study prepared for 
The National Institute of Education, Rreland and Stocking (1?74), 
(Project No. 3-0658, Grant No. NE~6-00~3-0116) 



the picture on the left is described as a " safety pen" and the picture on 
the right as a " fountain pen." But in both cases the word "pen" is pro- 
nounced as "pin." You will have to admit that both have sharp points! 
What I have just illustrated is a very subtle socio-cultural difference 
in the way the same ?.anguage is used. 

All cross-cultural instabilities in mental test items, however, are 
not due to socio-linguistic phenomena. Thorndike (1971) presented a 
hypothetical item about baking (the second item in Figure 1). Most house- 
wives would know that the correct answer is (c) . But most men, boys, and 
young girls not yet introduced to cooking would not. Thus, whether one is 
likely to know the correct answer is a function of his group membership. 
As Thorndike pointed out, whether such an item in unfair to a given group 
depends upon what it is being used for. If used in a test to select cooks, 
then one might find that the cooks selected were mostly women (assuming the 
test used a considerable number of similar items). 

The above two examples are rather obvious ones. Let me show you some- 
thing similar, but not obvious at all. Consider the third item in Figure 1. 
This is a vocabulary item in which & stem word is presented and the task is 
to identify the option most closely the same in meaning • In the investiga- 
tion I am about to report, 50% of groups of high school seniors identified 
as either North-Central Caucasians or Southern Caucasians responded (A) , 
the correct answer. But in a group of Latin-Americans living in the United 
States, and among whom half reported that English was not the language 
spoken most often in their homes, 56% responded correctly. All groups 
being compared were from carefully selected national random samples. A 
close look at this iten,, from a cross-cultural perspective, revealed that 



-3- 

both the stem word, convalesce, and the correct option, recuperate, have 
cognates in the Spanish language — convalecer and recuperar. 

I have presented three very different types of mental test Items, all 
of which demonstrate some kind of cross-cultural instability. The first 
item was created from my own experiences as a transplanted Southerner 
living and working in a foreign culture (New Jersey) . The second was 
obtained from a perceptive educational researcher. But the last item was 
discovered by means of a statistical procedure. 

This procedure, from Angoff and Ford (19 73) involves the cross-plotting 
of item difficulties for two different groups and generating a line of mutual 
regression through the resulting scatter of points. An example is pre- 
sented in Figure 2, where a national random sample of Oriental-Americans 
is compared to a random sample of North-Central American Caucasians. 
Deltas are obtained by the transformation 

A - 4Z + 13. 

The dotted line in Figure 2 is drawn at 45° as a line of reference. The 
solid line is the line of mutual regression (least squares are based on 
perpendicular distances). The item-points (+'s) represent most of the 
items from a six-test battery used in the Base-Year Survey of the National 
Longitudinal Study^ (Hilton & Rhett, 1973). The six tests were: Vocabulary, 
Picture-Number, Reading, Letter-Groups, Mathematics, and Mosaic Comparisons. 

Since the line of mutual regression falls above the dotted reference 
line, the suggestion is that the battery as a whole was easier for Oriental- 
Americans than it was for North-Central American Caucasians. What ifi of 
primary interest in the present paper, however, are the specific items that 

^ Conducted by ETS for the U. S. Office of Education. 
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were either easier or more difficult for a given group than other items in 
the battery. Some of the item-points are some distance from the solid line 
and thus are cause jr further inquiry. An indication of what characterizes 
the items that are relatively easier for Oriental-Americans is given by the 
computer-generated Table 1, which presents what are termed D-values for all 
items in a given subtest (in this case, Mathematics) and for all groups 
studied. The groups are identified in Table 1 as follows: AI-American 
Indian, AA-Af ro-American, MA-Mexican-American , PR-Puerto Rican, OL-other 
Latin-American, OR-Oriental-American, WE-White Eastern, WS-White Southern, 
and WW-White Western. The White North-Central group is not shown in the 
Table because it is the group with which all others are compared. 

The D-values represent the perpendicular distance, in delta units, from 
the line of mutual regression (the solid line in Figure 2). Negative D- 

^'alues represent distances above the solid line and positive D-valuer 
distance below the solid line. Therefore, negative D-values indicate 
that an item is relatively easier for the group plotted on the abscissa 
(in the case of Figure 2 and the column headed, or, in Table 1, Oriental- 
America!is). The relative ease of the Mathematics test, as a whole, in 
contrast to the entire six-test battery, as a whole, is suggested by 
the Group Mean at the bottom of Table 1. The mean D-value cf -.33 for 
Oriental-Americans would suggest that the Mathematics te&t vas easier 
for this group than other tests in the battery. The individual D-values 
indicate, however, that all items on the Mathematics sabtest were not 
easier for Oriental-Americans. Item 6 appears to have been substantially 
harrier. Item 16 was one of the easiest for Oriental-Americans. Consider 
the contrast of these two items : 
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Item 6 



Item 16 



(difficult for Oriental-Americans) 



(easy for Oriental-Americans) 



A 



B 



A 



B 



Degree rise in Degree rise in 
temperature from temperature from 

-6°F to +5°F ^5°F to +6°F 



1 
2 



1 

3 



2 3 



The task in this kind of item is to determine which of the following 
statements are true: (1) The quantity in Column A is greater, (2) The 
quantity in Column B is greater, (3) The two quantities are equal, or 
(4) The size relationship cannot be determined from the information 
given. For Item 6, even though 72% of the Oriental-Americans responded 
correctly with response (3), that the two quantities are equal, the De- 
values of Table 1 show that Item 6 was still relatively more difficult 
for theiTi than other items in the sjjc-test battery. All of the 

Caucasian groups either exceeded or approached 80% correct on Item 6. 

In contrast, 78% of Oriental-Americans correctly chose response (1) 
for Item 16, that the quantity in Column A is greater than the quantity 
in Column B. But only about 60% of the Caucasian groups responded 
correctly, with the White Eastern group slightly better than the White 
Western group (63% to 58%). One thing that characterizes the difference 
in these two items is the degree of verbalization. Item 6, which was 
relatively difficult for Oriental-Americans, has a lot of verbalization; 
Item 16 has none at all. In the entire Mathematics subtest, 12 of the 
25 items involved no verbalization: Items 1, 4, 5, 8, 10, 12, 13, 16, 



17, 19, 21, and 25. Looking at Table 1, it is seen that ten of these 
12 have large negative D-values for Oriental-Aajericane. Only Items 1 
and 4 do not fit this pattern. 

There is not time here to discuss a number of other interesting 
observations made using a procedure similar to that described. I will 
mention them briefly. The statistical procedures detected a set of 
items within uhe Reading subtest that appeared relatively easier for 
Blacks than other items in the battery. On investigation, these items 
ail were revealed to be associated with a reading passage about Black 
television. For minorities in general (with the exception of Oriental- 
Americans), the contrast of Item 2 and Item 5 in Table 1 is revealing. 
Item 2 dealt with money, something that might be learned outside of 
school. The large negative D-values for all minorities in Table 1 
suggest that Item 2 is relatively easier for them than most other items 
in the six-test battery. But Item 5 appears to be relatively more 
difficult for minorities. The large positive D-values for minorities 
in Table 1 contrast with negative D-values for all Caucasian groups and 
Oriental-Americans. What is so striking about this contrast is the 
simplicity of Item 5 — it merely asks to indicate which of the quantities, 
/9" or 9, is greater. While such an item is simple, it represents a type 
of knowledge usually only obtained in school. That minorities should 
perform so poorly on this type of item suggests that the schools they 
attend are providing very poor instruction in mathematics. While 72% of 
American Indians got Item 2 correct (which is greater, 11 dimes or $1.11), 
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Figure 1 

Examples of Cross-Culturally Unstable 
Mental Test Items 




The usual temperature for baking a cake is about: 
(A) 250° (B) 300O (C) 350O (D) 400° 



Select the one word or phrase whose meaning is clos 
the word in capital letters. 

CONVALESCE: 

(A) recuperate 

(B) bring together 

(C) condemn 

(D) appreciate 

(E) collect 



Figure 2 
Cross-plot of Deltas for Oriental 
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